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Abstract 

This study examines the lived experiences of students as expressed in their 
reflections on their experiences of learning at Ambrose University in Calgary. 
It uses quantitative outcomes-related data from the National Survey of Stu¬ 
dent Engagement and the Theological School Survey of Student Engagement 
to illuminate qualitative data obtained through student focus groups. The 
analysis of the qualitative data was conducted using the constant comparative 
method developed by Glaser and Strauss. The study concludes with recom¬ 
mendations for improving student engagement. 

Resume 

Cette etude examine les experiences vecues par des eleves telles qu’exprimees 
dans leurs reflexions sur leurs experiences d’apprentissage a l’Universite 
Ambrose, a Calgary. Afin d’eclairer les donnees qualitatives obtenues par le 
truchement de groupes de discussion d’etudiants, l’etude utilise les donnees 
liees aux resultats quantitatifs de l’Enquete nationale sur la participation 
etudiante (NSSE) et de l’Enquete de l’ecole de theologie sur l’engagement des 
etudiants (TSSSE). L’analyse des donnees qualitatives a ete realisee selon la 
methode comparative constante developpee par Glaser et Strauss. L’etude se 
conclut par des recommandations afin d’ameliorer l’engagement des eleves. 
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Many institutions of higher learning are under pressure from both internal and exter¬ 
nal stakeholders to report and clarity student learning outcomes. Faculty members, for ex¬ 
ample, want to know whether the curriculum is doing what it is supposed to be doing: that 
is, whether their students are actually learning (Astin & Lee, 2003), and whether they are 
teaching effectively (Chickering & Gamson, 1999; Scott & Scott, 2011). University adminis¬ 
trators and student development personnel see a close relationship between student reten¬ 
tion and financial benefit: that is, when students perceive that they have received a good ed¬ 
ucation, the university fulfills its mission and also acquires tuition income (Seidman, 2005). 

At the same time, the standards of outside stakeholders, such as regional and na¬ 
tional accrediting bodies, require institutions of higher learning to use both direct and 
indirect measures to demonstrate learning outcomes (Gordon, Ludlum, & Hoey, 2008). 
To provide the evidence that they are meeting their educational goals and fulfilling their 
mission, many institutions are beginning to analyze student engagement. One of the tools 
that colleges and universities use to measure student engagement is the National Survey 
of Student Engagement (NSSE), which has become the industry standard. Indeed, NSSE 
is a reliable means of both providing evidence of student learning and benchmarking it 
against that of peer institutions (Axelson & Flick, 2010; Gordon, Ludlum, & Hoey, 2008; 
Kahu, 2013). It calls attention to institutional strengths and deficiencies by rating stu¬ 
dents’ expectations in a certain area—say, the accessibility of faculty—along with their 
assessment of whether the institution has met their needs in this area. By taking steps 
to respond to concerns identified by NSSE results, institutions can improve the level of 
engagement of their students. According to Kuh (2003): 

The engagement premise is deceptively simple, even self-evident: The more students 
study a subject, the more they learn about it. Likewise, the more students practice 
and get feedback on their writing, analyzing, or problem solving, the more adept 
they become. The very act of being engaged also adds to the foundation of skills and 
dispositions that is essential to live a productive, satisfying life after college, (p. 25) 

NSSE depends on self-reporting, but its designers have attempted to address the prob¬ 
lems of validity associated with this kind of instrument by paying attention to five factors 
that are known to increase reliability: the familiarity of the question, clarity, currency of 
information, merit, and maintaining the privacy of the respondents (Pascarella, 2001). 
Although NSSE draws attention to an institution’s weaknesses in meeting its students’ 
expectations, it does not analyze them adequately from the perspective of the lived ex¬ 
periences of students, and so it has limited usefulness as a means of defining solutions. 
To compensate for this deficiency, the concerned institution must also take a qualitative 
approach to data collection. Qualitative researchers “emphasize the value-laden nature of 
inquiry. They seek answers to questions that stress how social experience is created and 
given meaning. In contrast, quantitative studies emphasize the measurement and analysis 
of causal relationships between variables, not processes” (Denzin & Lincoln, 2000, p. 8). 

In spite of NSSE’s heavy emphasis on quantitative data, both the researcher and the 
administration of Ambrose University agree that it is still, according to the literature, the 
best measure of student engagement for Ambrose. Both parties also realize, however, that 
NSSE results need to be complemented by qualitative data. 
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The purpose of this study is to compare the 2011 NSSE results from Ambrose under¬ 
graduate theology students and the 2011 Theological School Survey of Student Engage¬ 
ment 1 (TSSSE) results from Ambrose graduate students with the qualitative data gath¬ 
ered from two focus groups, one composed of undergraduate theology students and one 
composed of graduate theological students. The researcher has used both quantitative 
and qualitative approaches in the hope that the findings will suggest a course for future 
action on teaching, learning, and retention; but the researcher has chosen to give greater 
attention to the data supplied by the focus groups because they are based on the students’ 
own words and therefore provide a richer description of their experiences. The researcher 
hopes that the results of this study will make a modest contribution toward filling a gap 
in the literature about the engagement and learning of undergraduate and graduate stu¬ 
dents of theology. 


Context 

Ambrose University is a small liberal arts university and graduate theological school 
(fewer than 1,000 students) in Calgary, Alberta. Ambrose began to use NSSE in 2010 for 
undergraduate students and TSSSE in 2011 for graduate theological students. Early in 
each fall semester, the Student Development Department and the Office of Institutional 
Planning and Assessment report NSSE and TSSSE results to the faculty so that they may 
consider the implications for their teaching and for their students’ learning. 

As faculty members talked to the dean about the results of NSSE 2010, they were 
also concerned about student retention and about finding practical ways to improve their 
classroom practice. As a result of these concerns, the researcher began a conversation 
with the vice-president for student development to discuss further the results of the NSSE 
2010 survey and how it might be used to improve learning and retention. The researcher 
and the vice-president for student development decided that, if they expected to sup¬ 
ply the faculty with understandable and meaningful information, they would need to do 
a deeper analysis of students’ experiences than either the NSSE or the TSSSE statistics 
could provide. They therefore decided to follow up on the NSSE and TSSSE surveys the 
following spring with student focus groups in the fall of 2011. This initiative would also 
supply the vital information for the reaccreditation reviews of Ambrose’s undergraduate 
and graduate theological programs in the fall of 2012. 

Method and Participants 

This study used a mixed methodological approach known as a sequential study, in 
which “the researcher first conducts a qualitative phase of a study and then a quantita¬ 
tive phase, or vice versa. The two phases are separate” (Tashakkori & Teddlie, 1998, p. 
18). As mentioned, this study began with the quantitative phase in the spring of 2011 and 
concluded with the qualitative phase in the fall of 2011. Again, the focus of this article is 
on the qualitative phase. 

NSSE and TSSSE Participants 

In all, 129 randomly selected Ambrose undergraduate students responded to an invi¬ 
tation to fill out the NSSE survey in the spring of 2011. The response rate was 45%, which 
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was higher than the Canadian institutional average (32%) and the Alberta average (33%). 
Of Ambrose respondents, 81% were full-time students (nine or more credits) and 19% 
part-time (fewer than nine credits). NSSE only reports the means scores for first-year 
students and senior students. However, institution-reported class levels are used. Table 1 
compares selected characteristics of Ambrose respondents with those of other NSSE re¬ 
spondents in Canada, Alberta, and and the top 10% of all NSSE institutions (NSSE 2011). 

Table 1 

Characteristics of NSSE Respondents, 2011: Gender, Age, Enrolment Status, and Residence 



Ambrose (%) 

Canada(%) 

Alberta (%) 

NSSE 2011 ( 96 ) 

Gender 

Female 

64 

64 

65 

64 

Male 

36 

36 

35 

36 

Age 

24 or older 

20 

18 

20 

21 

Less than 24 

80 

82 

80 

79 

Enrolment status 

Full-time (9 or more credits) 

81 

89 

96 

89 

Part-time 

19 

11 

4 

11 

Place of residence 

On campus 

35 

20 

14 

32 

Off campus 

65 

80 

86 

68 


As Table 1 indicates, Ambrose respondents to the 2011 NSSE survey were similar in 
age and gender to NSSE respondents in Alberta, Canada, and the top 10% of all NSSE 
institutions. However, Ambrose had fewer full-time respondents and more respondents 
living on campus than the provincial, national, and the top 10% of all NSSE institution 
averages. 

For the 2011 TSSSE survey, 50 randomly selected graduate students from Ambrose 
responded, for a response rate of 40%. Data for comparison with other institutions were 
not available, but Table 2 provides internal comparisons of respondents’ characteristics. 

Focus Group Participants 

According to Krueger and Casey (2009), focus groups are most helpful when “the pur¬ 
pose is to uncover factors that influence opinion, behavior or motivation. Focus groups 
can provide insight into complicated topics when opinions or attitudes are conditional 
or when the area of concern relates to multifaceted behavior or motivation” (p. 19). The 
semi structured focus groups in this study met over several weeks in October and Novem¬ 
ber 2011. They consisted of undergraduate and graduate theological students who had 
responded to an email invitation circulated by the Student Development Department. 
The undergraduate theology focus group consisted of three women and three men. Five 
were full-time students (nine or more credits), and one was part-time. Five were under 24 
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years of age, and one was older than 24. Three students lived in residence and three lived 
off campus. The graduate theology focus group had seven students, four men and three 
women. Three were full-time students, and four were part-time. Five were 31 years old or 
older, and two were under 31. Only one of the seven lived on campus. 

Table 2 

Characteristics of Ambrose Graduate Students Responding to TSSSE, 2011: Gender, 
Age, Enrolment Status, and Residence 


Year of Study 

First year 
( 96 ) 

Second 
year(%) 

Third year 
( 96 ) 

Fourth 
year(%) 

Other (%) 

Total ( 96 ) 


26 

30 

24 

4 

16 


Gender 

Female 

39 

27 

33 

0 

62 

36 

Male 

61 

73 

67 

100 

38 

64 

Age 

31 or older 

62 

87 

42 

100 

100 

72 

Less than 31 

38 

0 

42 

0 

0 

20 

No response 

0 

13 

16 

0 

0 

8 

Enrolment status 

Full-time (9 or 

46 

67 

42 

0 

12 

44 

more credits) 

Part-time 

54 

33 

58 

100 

88 

56 

Place of residence 

On campus 

0 

0 

8 

0 

0 

2 

Off campus 

100 

100 

92 

100 

100 

98 


The characteristics of the undergraduate theology focus group were very similar to 
those of the Ambrose students who had completed the NSSE survey, except in gender and 
place of residence. In the focus group, women were 50% of the participants; among the 
Ambrose NSSE participants, 64% were women. The researcher ensured that 50% of the 
undergraduate students in the focus group lived on campus, because the Ambrose NSSE 
comparison group included a relatively high proportion of on-campus students (35% vs. 
the Canadian average of 20% and the Alberta average of 14%). 

The characteristics of the graduate theology focus group resembled those of the Am¬ 
brose TSSSE respondents, except that 14% percent of the participants lived in residence 
whereas only 2% of the Ambrose TSSSE respondents did. The percentage of female par¬ 
ticipants was also higher in the focus group (43% vs. 36%). 

The researcher used only two focus groups because the students’ characteristics were 
very similar to those of the Ambrose NSSE and TSSSE respondents. According to Patton 
(2002), “The validity, meaningfulness, and insights generated from qualitative inquiry 
have more to do with the information-richness of the cases selected and the observation¬ 
al/analytical capabilities of the researcher than with the sample size (p. 245).” 
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The researcher met individually on campus with each focus group for about two hours. 
At the beginning of each meeting, the researcher had the participants sign a consent form 
that indicated the purpose of the study and how the data would be used. The researcher 
also received oral and written consent from the participants in both focus groups to video¬ 
record the sessions and transcribe the discussion. 

The interview guides for the focus groups contained 11 questions each. The questions 
were mainly based on those areas of engagement in which Ambrose respondents to the 
NSSE and TSSSE surveys scored lower than their peers. The researcher decided to use 
these deficiencies as a point of departure because he wanted to increase his understand¬ 
ing of the factors that promote student engagement. Along with the questions, the re¬ 
searcher used probes to stimulate the participants to be more introspective and to share 
more deeply (Glesne & Peshkin, 1992). The researcher emailed the final transcripts to 
each student for member checking, so that they could personally verily the viewpoints 
they had expressed and the statements they had made and thereby capture more fully 
their unique experiences (Gall, Gall, & Borg, 2003). Some students responded with sim¬ 
ple affirmation, others responded with corrections to the final transcript, and still others 
did not respond at all. 

For the analysis of the focus group data, the researcher used the constant comparative 
method (CCM) developed by Glaser and Strauss (Glaser & Strauss, 1967; Strauss, 1987; 
Glaser, 1992), along with a six-step process for using the CCM that was developed by 
Savin-Baden and Howell-Major (2013): 

• Identify categories in events and behaviour. 

• Name indicators in passage and code them (open coding). 

• Continually compare codes and passages with those already coded to find consis¬ 
tencies and differences. 

• Examine consistencies or patterns between codes to reveal categories. 

• Continue the process until the category “saturates,” and no new codes related to it 
are identified. 

• Determine which categories are the central focus (axial categories) and, thus, form 
the category, (p. 437) 

Charmaz (2005) states that the CCM of grounded theory involves 

• comparing different people; 

• comparing data from the same individuals with themselves at different points in 
time; 

• comparing incident with incident; 

• comparing data with category; and 

• comparing one category with another, (p. 515) 

The researcher modified the steps used by Savin-Baden and Howell-Major (especially 
bullet points two and six ) because both NSSE and TSSSE already provided categories, 
called benchmarks, of empirically tested sets of practices that cause students to experi¬ 
ence satisfaction in the experiences of learning (Kuh, 2009; Lerer & Talley, 2010). In 
other words, the coding of student experiences from the focus group used the already pre¬ 
determined categories of NSSE and TSSSE. The researcher made comparisons between 
the NSSE and TSSSE results and the data collected from the focus groups. For both the 
undergraduate and the graduate theology student experiences, the researcher compared 
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data with category and with individual responses, and also compared one category with 
another. Moreover, colleague checking was used for the codes for reliability and triangu¬ 
lation purposes. Reliability means consistency in the use of the data between colleagues 
(Richards, 2011). The method used was to compare two codings of the same data. 

Findings 

The researcher organized findings under the five NSSE/TSSSE benchmarks 2 of effec¬ 
tive educational practice: student-faculty interaction, supportive campus environment, 
enriching educational experiences, level of academic challenge, and active and collabora¬ 
tive learning. The NSSE and TSSSE scores in the Tables have been given by the mean 
and the demographic characteristics have been given in percentages. The mean is the 
weighted arithmetical average of the student-level benchmark scores. Effect size indicates 
the practical significance of the mean difference. It is calculated by dividing the mean dif¬ 
ference by the pooled standard deviation. In practice, an effect size of .2 is often consid¬ 
ered small, .5 moderate, and .8 large. A positive sign value indicates that the institution’s 
mean was greater than that of the comparison group, thus showing an affirmative result 
for the institution. A negative sign indicates the institution lags behind the comparison 
group, suggesting that the student behaviour or institutional practice represented by the 
item may warrant attention. The researcher will examine below in detail the scores that 
have negative signs. 

Undergraduate Theology Students 

After examining the NSSE benchmark comparisons, the researcher decided to exam¬ 
ine closely two groups of survey findings with comparatively low scores: results for the en¬ 
riching educational experiences and active and collaborative learning benchmarks. Am¬ 
brose lagged behind its peer institutions in these two benchmarks. This was then followed 
by consideration of the student-faculty interaction and supportive campus environment 
benchmarks. Since the undergraduate theology focus group respondents said little about 
the level of academic challenge, this category was not given extensive treatment. 

Enriching educational experiences (EEE). NSSE defined this benchmark as 
complementary learning opportunities that enhance academic programs: diverse ex¬ 
periences that teach students valuable things about themselves and others. Technology 
facilitates these experiences by enhancing collaboration between peers and instructors. 
Internships, community service, and senior capstone courses provide opportunities to 
integrate and apply knowledge. The survey questions in this category measured indica¬ 
tors such as these: 

• hours spent participating in school-sponsored activities (e.g., student life events, 
chapel services, spiritual formation groups, and mission endeavours); 

• practicum or field experience; 

• community service or volunteer work; 

• cross-cultural experience; 

• foreign-language coursework; 

• independent or directed study; 

• culminating senior experience (e.g., capstone courses, senior projects or theses, and 
comprehensive exams); 
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• serious conversations with students of a different race or ethnicity; 

• serious conversations with students who are very different from the respondent in 
terms of theological convictions, political opinions, or personal values; and 

• whether the campus environment encourages contact among students from differ¬ 
ent economic, social, religious, and racial or ethnic backgrounds. 

• using electronic medium (e.g., listserv, chat group, Internet, instant messaging, 
etc.) to discuss or complete an assignment. 

Table 3 

Mean Comparisons for Enriching Educational Experiences Benchmark, NSSE 2011 



Ambrose 

Canada 

Effect size 

Alberta 

Effect size 

NSSE 2011 

Effect size 

First-year 

students 

20.8 

24.1 

-.27 

24.1 

-.27 

27.0 

-.46 

Senior 

students 

36.7 

34-3 

•15 

35-4 

.08 

39-5 

-•15 


The NSSE 2011 comparison results are shown in Table 3. They suggest that Ambrose 
needed to focus on this particular benchmark because the effect size of -.46 for first-year 
students indicates that Ambrose’s overall EEE lagged behind that of its peer institutions. 
However, the undergraduate theology students in the focus group mildly disagreed with 
this finding because they had benefited from field experiences in their courses and from 
practicums and internships. One student put it this way: “The practical application in 
class where we discussed a topic and then went on a field trip really helped my learning.” 
In addition, many students in the focus group had taken travel-study courses. One such 
student found it “cool” to have received academic credit for a course that had enhanced 
learning. In summary, the students in the focus group believed that learning happens 
when they feel engaged in the learning process and that they had had this sense of engage¬ 
ment in many of their classes. 

The members of the focus group also indicated that Ambrose provided many opportu¬ 
nities for co curricular activities that enriched their educational experience. Many of the co 
curricular activities provided opportunities for conversations about life issues and person¬ 
al values. However, the members of the group also mentioned that they had to take the ini¬ 
tiative to participate in these activities if they expected to reap the benefits of involvement. 

As mentioned above, Ambrose had a higher percentage of students living in residence 
than the average Alberta, Canadian, or the top 10% of all NSSE institutions, so the re¬ 
searcher concluded that it was important to understand the perspective of off-campus stu¬ 
dents with respect to on-campus participation. The three residence and three off-campus 
students in the focus group all acknowledged that it was harder (but not impossible) for 
off-campus students to be involved in on-campus co curricular activities. One off-campus 
student summarized the sentiments of the three off-campus students as follows: “I need 
to work, and although I want to come I don’t have time_Besides that, it’s an extra trip.” 

A number of students in the focus group, as stated earlier, indicated that they had no 
lack of co curricular opportunities on campus. However, they found it hard to balance 
their co curricular involvements with the demands of their academic program. Their un- 
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ease was made more intense by the implicit pressure they felt to be involved in as many 
co curricular activities as possible. The group suggested that the administration find ways 
of helping students to balance these competing expectations. 

Ambrose scored low on the EEE section that dealt with the faculty’s use of electronic 
media. Such media help students collaborate with one another and with the instructor. 
The focus group concurred with the Ambrose NSSE finding (mean score of 2.40), noting 
that some professors used Moodle (a learning management system) effectively, while oth¬ 
ers did not use it at all. 

With respect to the survey question on “culminating senior experience,” the two se¬ 
niors in the focus group regarded it as being inapplicable to their situation, because their 
degrees were more professionally oriented and did not require any kind of senior project. 

Active and collaborative learning (ACL). This benchmark assumed that stu¬ 
dents learn more when they are intensely involved in their education and when they are 
asked to think about what they are learning in different settings. Collaborating with oth¬ 
ers in solving problems or mastering difficult material contributes to this sense of involve¬ 
ment and prepares students for the messy, unscripted problems they will encounter daily 
during and after their studies. The survey questions for the ACL benchmark included 
these components: 

• asking questions in class or contributing to class discussions; 

• making a class presentation; 

• working with other students on projects during class; 

• working with classmates outside of class to prepare class assignments; 

• participating in a community service project or ministry endeavour as part of a 
course or for academic credit; 

• discussing ideas from your readings or classes with others outside of class (stu¬ 
dents, family members, co-workers, etc.); and 

• hours spent working in a ministry-related position. 

The undergraduate theology students in the focus group said that a lot of their 
classes were discussion-based. One student put it this way: “I learn best in discussion, it 
hits it home for me a bit more. ... It helps me make sense of my learning—it’s not just 
knowledge for exams.” 

Table 4 

Mean Comparisons for Questioning and Discussion Component of Active and Collab¬ 
orative Learning Benchmark, NSSE 2011 




Ambrose 

Canada 

Effect 

size 

Alberta 

Effect 

size 

NSSE 

2011 

Effect 

size 

Asking questions 
in class or con- 

First-year 

students 

2-45 

2.32 

•14 

2.30 

•17 

2.72 

-• 3 i 

tributing to class 
discussions 

Senior 

students 

3.12 

2.74 

•41 

2.72 

•43 

3-04 

.09 
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Ambrose had above-average scores in one ACL question: as shown in Table 4, Am¬ 
brose was doing better than other schools in Canada and Alberta with respect to asking 
questions in class and classroom discussion. Senior students did even better, exceeding 
even the the top 10% of all NSSE institution scores for this item. One senior student in 
the focus group had this to say about classroom discussions: “I actually find that I build 
friends from discussion with my classmates. I get to know people through what they say 
in class, and I find that becomes very real because we’re allowed to express how the topic 
relates to us.” First-year students tend not to speak up or to ask questions as much as se¬ 
nior students. All the mean scores indicate this. Part of the reason is that first-year classes 
tend to be bigger than upper-level classes. The first-year students in the focus group said 
that they did not speak up in class or ask questions because they were unfamiliar with 
the subject matter and felt it was best to listen. One senior student, reflecting on his first 
year, observed, “Sometimes professors rein you in because you’re giving an uneducated 
response.. . . You do need the foundations to get to the real stuff. . . . Although founda¬ 
tions are important they can be boring.” 

Table 5 

Mean Comparisons for Group Work Component of Active and Collaborative Learning 
Benchmark, NSSE 2011s 




Ambrose 

Canada 

Effect 

size 

Alberta 

Effect 

size 

NSSE 

2011 

Effect 

size 

Working with 
other students 

First-year 

students 

1-9 

2.13 

-.22 

2.19 

-.29 

2.38 

-■51 

on projects 
during class 

Senior 

students 

1-7 

2.23 

-•53 

2.26 

-.58 

2.51 

-.84 


Ambrose students did not score as well on the question about working with class¬ 
mates on projects during class. The researcher posed to the focus group the NSSE ques¬ 
tions whose results are summarized in Table 5: “To what extent do you feel working with 
other students on group assignments inside and outside of the classroom would enhance 
your learning? What disadvantages do you see in doing group assignments together and 
how might they be addressed?” The students responded that they had concerns about 
group work, and these concerns (if they were shared by Ambrose respondents as a whole) 
may have accounted for Ambrose’s low NSSE scores. Many of the students in the focus 
group thought that their instructors did not take into account such factors as differences 
in learning styles or differences in personality when they assigned group projects. In any 
event, the group observed that a few people in the work group did the majority of the 
work. As one student put it, “Some in the group didn’t carry their weight for the class as¬ 
signment.” The participants suggested that group work would become more meaningful 
if instructors were to provide several options for group projects, and if they were to take 
into account personality and learning styles in assigning groups. 

Student-faculty interaction (SFI). This benchmark had to do with how students 
learn first-hand by interacting with professors inside and outside the classroom and by 
observing how faculty-as-experts think about and solve practical problems. In the pro- 
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cess, professors may become role models, mentors, and guides for continuous, lifelong 
learning. The SFI items included these indicators: 

• discussing grades or assignments with an instructor; 

• talking about career plans with a faculty member or advisor; 

• discussing ideas from your readings or classes with faculty members outside of 
class; 

• working with faculty members on activities other than coursework (committees, 
orientation, student-life activities, etc.); 

• receiving prompt written or oral feedback from faculty on your academic perfor¬ 
mance; and 

• working on a research project with a faculty member outside of course or program 
requirements. 

On the question that dealt with student-faculty interaction outside of class time on 
ideas from readings or classes, Ambrose first-year students did better than their peers in 
schools in Canada and Alberta. Senior students also exceeded not only the mean Alberta 
and Canada scores but also the top 10% of all NSSE institution scores. The focus group 
agreed with this finding and attributed Ambrose’s good showing to small class size and 
the willingness of faculty to discuss further or clarity ideas presented in class or in the 
readings. All participants thought that their instructors went the extra mile in helping 
their students understand the subject matter. Some also mentioned that they had talked 
with faculty about vocational matters. Others confessed that they had left this resource 
untapped. For example, many of the first-year students in the focus group seemed to be 
unaware that their academic advisors could help them navigate course sequencing or gain 
a comprehensive understanding of the curriculum. These findings suggest communica¬ 
tion between students, especially first-year students, and faculty needed to improve. 

Supportive campus environment (SCE). This benchmark assumed that students 
perform better and are more satisfied at universities that are committed to their success 
and cultivate positive working and social relations among different groups on campus. 
The SCE-related survey questions covered the following indicators: 

• campus environment that provides the support needed to help succeed academi¬ 
cally; 

• campus environment that helps students cope with their non-academic responsi¬ 
bilities (work, family, etc.); 

• campus environment that provides the support needed to thrive socially; 

• quality of relationships with other students; 

• quality of relationships with faculty members; and 

• quality of relationships with administrative personnel and offices. 

All students in the focus group had attended the orientation at the beginning of their 
first fall semester. They described it as a welcoming and enjoyable experience because, 
thanks to the involvement of upper-year orientation leaders, it gave them the opportu¬ 
nity to meet other students. The orientation period also provided them with helpful sug¬ 
gestions for navigating university life. The students in the focus group also appreciated 
the variety of co curricular activities available to those willing to initiate the necessary 
contact. All spoke positively about residence assistants and floor meetings, both of which 
helped them meet social and spiritual needs. They also valued the high quality of their re- 
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lationship with faculty. Some said that they went to faculty not just for academic advising 
but also to seek advice on personal and spiritual matters. The members of the focus group 
also availed themselves of the counselling services offered by the Student Development 
Department. They also mentioned that they found it easy to get spiritual and personal 
help on campus. 

However, group members expressed a concern that commuter students needed to be 
“super-intentional” in developing relationships. Some mentioned that off-campus stu¬ 
dents found it easier to befriend other off-campus students than to develop relationships 
with residence students. According to one residence student, “We are with each other 
24/7 and sometimes we forget to include others not in residence.” 

Graduate Theology Students 

TSSSE, the survey instrument that measures the engagement levels of graduate theol¬ 
ogy students, used the same indicators, definitions, and terms as NSSE. Four out of the 
five NSSE benchmarks have been defined above; the fifth, level of academic challenge, 
will be elaborated on below. Further, not only are the mean scores reported for first-year 
students and senior graduate theology students but also institution-reported class levels 
are used and reported because master degrees are typically two or three years in duration. 
Thus, all randomly selected or census-administered students are included in the analysis. 

Enriching educational experiences (EEE). The mean score for EEE in the TSSSE 
2011 results for Ambrose first-year graduate theology students was 21.1; for peers at other 
institutions, it was 28.7. For senior students, the mean scores were 37.7 and 41.0 respec¬ 
tively. The TSSSE results indicate that some graduate theology students were missing the 
benefits of full participation in the life of the campus by choosing not to attend events or 
engage in co curricular opportunities. The focus group helped identify the primary moti¬ 
vation of those who participated, the barriers for those who did not, and the perceptions 
of the value of participation in activities outside of the classroom (the discussion in the 
next section below will elaborate further on this). 

Most of the graduate theology students in the focus group valued the spiritual for¬ 
mation component of their educational experience and believed that aspects of spiritual 
formation enriched their educational experience. Their primary motivation for involving 
themselves in spiritual formation was to meet others and to find and build community. 
A number of them mentioned that they or their fellow students have other communities 
for personal support, and that these other communities also enhanced their educational 
experience. One student put it this way: “My primary community that supports me is my 
church, but it’s nice that there are other students who can more specifically understand 
the journey of graduate education at Ambrose.” 

Supportive campus environment (SCE) and student-faculty interaction 
(SFI). The SCE mean scores for first-year students were 50.1 (for Ambrose) and 57.3 
(for TSSSE 2011 respondents overall); for senior students they were 62.8 (Ambrose) and 
51.7 (TSSSE). These scores indicate that, contrary to the experience of their TSSSE peers, 
Ambrose graduate theology students’ perception that their campus environment was sup¬ 
portive increased over the years. When asked “To what extent do you feel that you and 
your success are a high priority for the staff and faculty who work at Ambrose?” most 
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students in the focus group answered, “The faculty places a high priority on our success.” 
One student offered an explanation: “If my professor didn’t mention the student services 
in class, I wouldn’t have used it.” 

The service that the students in the focus group used the most was Learning and In¬ 
ternational Services. They believed that the director of this department was determined 
to help them succeed academically. They also mentioned that the Finance Office and the 
Registrar’s Office served them efficiently and professionally. Although graduate theol¬ 
ogy students are older than undergraduate students, they still need both community and 
physical space to build deep relationships with their fellow students. Most students in 
the group believed that Ambrose offered student services only to undergraduates. Some 
thought that they would like to use these services but were not sure whether they could 
legitimately do so. A number of them echoed the sentiment of the student who said, “I feel 
guilt in using student service because I might be wasting the person’s time or I’m taking 
time that an undergrad student could use.” Another student added, “I wanted to use the 
career services but I wasn’t sure it was for me to use.” Ambrose clearly needed to address 
this misperception. 

When asked about the physical layout of the campus, the members of the focus group 
responded that they liked the existing campus but that they initially found it hard to navi¬ 
gate. The maps they were provided with did not help because the terminology the maps 
used was incomprehensible. 

The 2011 SFI mean scores (Ambrose first-year students, 22.2; TSSSE first-year stu¬ 
dents, 21.7; Ambrose senior students, 32.0; TSSSE senior students, 27.1) showed that 
Ambrose was keeping up with or slightly exceeding its comparison group in this area. The 
focus group interviews indicated that graduate theology students sought support from 
two sources: other students and their faculty advisors. All the students in the focus group 
appreciated the presence of faculty at community events, their desire to build community, 
and their helpfulness in providing academic and personal guidance. A number of stu¬ 
dents in the focus group mentioned that they had discussed vocational issues with faculty. 

Level of academic challenge (LAC). Challenging intellectual and creative work is 
central to student learning and collegiate quality. Universities promote high levels of stu¬ 
dent achievement by emphasizing the importance of academic effort and setting high expec¬ 
tations for student performance. LAC-related survey questions covered such areas as these: 

• hours spent preparing for class (studying, reading, writing, doing homework or lab 
work, etc., related to academic program); 

• number of assigned textbooks, books, or book-length packs of course readings; 

• number of written papers or reports of 20 pages or more, between five and 19 pages, 
and fewer than five pages; 

• coursework that emphasizes analysis of the basic elements of an idea, experience 
or theory; 

• coursework that emphasizes synthesis and organizing of ideas, information, or ex¬ 
periences into new, more complex interpretations and relationships; 

• coursework that emphasizes making of judgments about the value of information, 
arguments, or methods; 

• coursework that emphasizes applying theories or concepts to practical problems or 
in new situations; 
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• working harder than you thought you could to meet an instructor’s standards or 
expectations; and 

• campus environment that emphasizes spending significant amount of time study¬ 
ing and on academic work. 

Table 6 

Mean Comparisons for Level of Academic Challenge Benchmark, TSSSE 2011 



Ambrose 

TSSSE 2011 

First-year students 

73-0 

55-2 

Second-year students 

79-8 

554 

Senior students 

754 

54-2 


As Table 6 indicates, the mean score of Ambrose first-year graduate theology students 
on this part of the survey was 73.0; the mean score for TSSSE respondents as a whole was 
55.2. The first-year students in the focus group indicated that they did spend consider¬ 
able time preparing for class and doing homework and assignments. Full-time students 
especially seemed to spend a significant amount of time studying; as mentioned above, 
this diligence might have hindered their ability to participate in social events and other 
community building activities. 

Table 7 

Mean Comparisons for Active and Collaborative Learning Benchmark, TSSSE 2011 



Ambrose 

TSSSE 2011 

First-year students 

32.5 

33-2 

Second-year students 

37-9 

384 

Senior students 

40.5 

39-0 


Active and collaborative learning (ACL). Table 7 shows the overall mean com¬ 
parison with respect to ACL: Ambrose’s mean scores are similar to those of TSSSE re¬ 
spondents as a whole. Students in the focus group did not discuss to any significant degree 
this aspect of their student experience. However, they did mention collaborative learning 
activities that took place outside of the classroom in such places as the graduate lounge 
and the group study rooms in the library. As mentioned above, they did not believe that 
the graduate meeting area provided enough privacy for small-group conversation; and 
even though they enjoyed conversing in the group study rooms, they thought they were 
monopolizing the space. 

Both the overall TSSSE results and the focus group interviews indicated that graduate 
theology students believed that working with other students inside and outside of the class¬ 
room enhances learning. Two items from the TSSSE ACL-related questions made the point: 
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• working with other students on project during class; and 

• working with classmates outside of class to prepare class assignments. 

The answer options were sometimes, often, and very often. According to the raw 
data, Ambrose first-year graduate students responded that they worked with others dur¬ 
ing class 61.6% of the time; senior students said they did so 75% of the time. Ambrose 
first-year graduate students said they worked with others outside of class 53.9% of the 
time; senior students said they did this 66.7% of the time. 

Most members of the focus group, when asked “Where have some of your positive 
educational engagement experiences happened inside and outside of classes?” responded 
that working with others helped in their learning. One senior student said: “One or two of 
the professors have group assignments and that helps me, especially when I don’t under¬ 
stand something.” 


Discussion 

Undergraduate Theology Students 

Ambrose’s mean overall comparison score for EEE falls below the provincial and na¬ 
tional benchmarks for first-year students and slightly above the provincial and national 
scores for senior students. The student focus groups provided an explanation for this find¬ 
ing. Indeed, they slightly disagreed with one aspect of it. They indicated that professors in 
their classroom teaching provided a lot of EEE and that students had ample opportunity 
to connect with one another by way of the co curricular activities on campus. However, 
they also identified barriers to EEE: namely, travel and lack of time. Both residence and 
off-campus students mentioned lack of time, because of either part-time work or academic 
studies, as the main deterrent to participation in co curricular activities. Many respondents 
voiced a concern for off-campus students who had to travel back to campus for events or 
who had to stay longer on campus to attend them. The matter of commuter students’ lack 
of connection with other students also came up in the focus groups’ discussion of SCE. This 
sense of disconnection is commonly felt by commuter students at all institutions, and it 
has received considerable attention in the literature (Alfano & Eduljee, 2013; Astin, 1984; 
Bradley, 2006; Kuh, Gonyea, & Palmer, 2001; Newbold, Mehta, & Forbus, 2011). 

The ACL narratives from the student focus group confirmed the NSSE finding that 
Ambrose was on par with or lagging a bit behind its provincial and national counterparts 
in this area. Ambrose’s chief weakness in ACL had to do with group work. The student 
focus group feedback indicated at least one reason for this: students did not like group 
work because there is no way to measure the relative contribution of each group member, 
and because professors did not know how to use group work effectively. As both the NSSE 
results and the focus group discussions indicated, Ambrose actually did well in other as¬ 
pects of ACL, such as asking questions and classroom discussion. These data aligned well 
with the findings of teaching and learning and student engagement literature, which indi¬ 
cate that instructors can foster engagement in a variety of ways, among them discussion 
in classes, knowing students’ names, responsiveness to students’ questions, urging stu¬ 
dents to seek help, and assigning effective aids to learning (Ambrose et.al, 2010; Barkley, 
2010; Miller, 2008). 
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Students in the focus group spoke very little about the level of academic challenge, 
even though some of the focus group questions targeted this category. The researcher 
found this rather disconcerting, as there was little conversation about the teaching and 
learning moments in the classroom, course objectives, or the nature of assignments. The 
researcher discovered a partial explanation for this deficiency on reviewing the overall 
NSSE 2010 results for LAC (Table 8). The scores for Ambrose met or exceeded those of 
institutions in Canada in 2010, but in 2011 the Ambrose first-year scores fell slightly be¬ 
low the national and provincial averages. The mean comparison scores for 2011 (Tables 9 
and 10) indicate more precisely the single items in which Ambrose needed to increase the 
level of academic challenge. With the exception of the results for campus environment, 
Ambrose’s single-item scores followed the provincial and national trends in that they in¬ 
creased from the first year to the senior year. Ambrose’s statistically significant areas of 
weakness vis-a-vis national and provincial results were the following items: 

• for first-year students: analyzing the basic elements of an idea, experience, or the¬ 
ory, such as examining a particular case or situation in depth and considering its 
components; 

• for both first-year and senior students: number of written papers or reports of 20 
pages or more; and 

• for senior students: number of written papers or reports of between five and 19 pages. 
Table 8 

Mean Comparisons for NSSE Level of Academic Challenge (LAC), 2010-2011 
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First-Year 

51-1 

50.9 

.01 

57-2 

-•47 

53-9 

-.20 

48.2 

51-2 

-•23 

50.6 

-.19 

53-2 

-•38 

Senior 

56.3 

56.6 

-•03 

60.4 

-•30 

57-5 

-.09 

55-7 

55-5 

.02 

54-3 

.10 

57-1 

-.10 


Table 9 

Mean Comparisons for Analyzing an Idea, Experience, or 
of Academic Challenge Benchmark, NSSE 2011 

Theory Component of Level 

Ambrose Canada Effect Alberta 

size 

Effect 

size 

NSSE 

2011 

Effect 

size 

Analyzing the basic elements of an idea, experience, or theory, such as examining a 
case or situation in depth and considering its components 

First-year students 2.87 3.11 -.31 3.14 -.35 3.16 

particular 

-•38 

Senior students 3.20 3.25 -.07 3.25 

-.08 

3-30 

-.14 
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Table 10 

Mean Comparisons for Written Assignments Components of Level of Academic Chal¬ 
lenge Benchmark, NSSE 2011 


Ambrose Canada 

Effect 

size 

Alberta 

Effect 

size 

NSSE 

2011 

Effect 

size 

Number of written papers or reports of 20 pages or more 




First-year students 1.11 1.41 

-.40 

1.29 

-.28 

1.30 

-.27 

Senior students 1.36 1.81 

-•53 

1.76 

-.48 

1.65 

-•36 

Number of written papers or reports between 5 and 19 pages 




First-year students 2.34 2.26 

.10 

2.15 

•25 

2.21 

-.16 

Senior students 3.15 2.70 

•48 

2.58 

•63 

2.56 

-.62 


From 2010 to 2011, Ambrose’s mean single-item scores for these three components 
fell slightly. For example, the first-year mean score for analyzing the basic elements of an 
idea, experience, or theory slipped from 2.98 to 2.87 (Table 11). Over the same period, se¬ 
nior students’ scores for this item improved slightly, from 3.19 to 3.20. The faculty of Am¬ 
brose needed to devise a long-term strategy for increasing the level of academic challenge. 

Table 11 

Mean Comparisons for Analyzing an Idea, Experience, or Theory Component of Level 
of Academic Challenge Benchmark, NSSE 2010 and NSSE 2011 



2010 

2011 

Analyzing the basic elements of an idea, experience, or theory, such as examining a particular 
case or situation in depth and considering its components 

First-year students 

2.98 

2.87 

Senior students 

3-19 

3-20 


Graduate Theology Students 

Ambrose’s TSSSE 2011 results seemed to be consistent with the feedback from the 
graduate student focus groups. However, a few inconsistencies occurred with respect to 
EEE. Enriching educational experiences really depends on the degree to which students 
engage with one another. Engagement, in turn, has a direct correlation with how often 
students are on campus. Obviously, the full-time students felt more engaged. Like their 
undergraduate counterparts, graduate theology students attributed their lack of time 
spent on campus to the demands of coursework. “Work, lots of hard studying!” was how 
one student summed up the Ambrose experience. However, graduate theology students 
also mentioned other time-limiting factors, such as full-time employment and family. 
This is understandable because most graduate theology students were studying part-time 
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and taking fewer than nine credits. The 2011 enrolment figures from the Registrar’s Office 
confirmed this: 88 (69%) were studying part-time, and 40 (31%) were full-time. Indeed, 
most graduate theology students in North America are studying part-time (Aleshire, 
2008; Association of Theological Schools, 2011-2012). 

Another inconsistency with respect to EEE centred on space and communication. All 
the students in the focus group mentioned two primary hindrances to building commu¬ 
nity, the first being the lack of common space for in-depth conversation. Graduate theol¬ 
ogy students had only a couple of tables in a hallway as their meeting space, which led 
one student to complain that “the space we have now feels like an add-on, it’s not private 
enough.” Another half-jokingly remarked, “There are two places you can talk theology, 
at the bar and here at Ambrose. You can’t really talk theology in the church.... Here on 
campus we’re allowed to ask questions and ponder and don’t feel like we have to have all 
the answers. Having a good designated space would enhance and welcome conversation 
from a diverse group of people in age, culture, and denominational backgrounds.” 

The communication of social events is another aspect of enhancing social life that 
was being done poorly. One student had this to say: “You have to look for things, it’s not 
communicated; or if it is, it needs to be communicated multiple times!” Another student 
added, “I found out about the event only after the fact.” Most members of the focus group 
agreed that the publicizing of community-building events needed to be improved. 

Recommendations for Moving Forward 

The following recommendations are based on the narratives from the two focus groups: 

• That a collegium for undergraduate commuter students be established to facilitate 
the following educational outcomes: (1) enhanced community within the student 
body and between students and faculty; (2) increased opportunities for individual 
and group study; and (3) social integration via the sharing of meals. 

• That the director of commuter life oversee the development of a commuter life pro¬ 
gram (to facilitate further the development of community). 

• That a dedicated space for graduate theology students be found and developed ap¬ 
propriately to enhance (1) community life within the student body and between stu¬ 
dents and faculty; (2) opportunities for individual and group study and theological 
conversations; and (3) social integration through the sharing of meals. 

• That both the Student Development Department and the faculty find ways to assist 
students in balancing academic and cocurricular expectations. 

• That the faculty, through the Teaching and Learning Committee, consider ways to 
enhance active learning in the classroom and group learning. 

• That the faculty devise strategies to increase the level of academic challenge. 

Limitations 

The findings and recommendations of this preliminary study are limited to a reli¬ 
giously based university and graduate theology school in western Canada. The findings 
and recommendations would likely be different at other institutions of higher learning in 
other locations and in settings of greater diversity. If other aspects of engagement were 
measured by the NSSE and TSSSE surveys, and if other aspects of the focus group in- 
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terviews had been emphasized, the results might have differed or changed. The small 
number of focus groups was also a limiting factor; a greater number of groups might 
have affected the findings. Moreover, because this study is qualitative in nature and not 
quantitative, it would require considerable refinement to make its findings generalizable. 
Finally, this study is exploratory in nature, and so it should not be used on its own as a 
means of explaining the intricacies and complexities of student engagement. 

Conclusion 

Ambrose has used the NSSE and TSSSE surveys as means of improving student learn¬ 
ing inside and outside the classroom. Kuh’s (2003) comment about the use of NSSE ac¬ 
cords well with this intention: “Although NSSE does not directly assess learning outcomes, 
the results from the survey point to areas where colleges are performing well in enhancing 
learning, as well as to aspects of the undergraduate experience that could be improved” (p. 
26). Kuh’s view applies equally to the graduate experience as well. For Ambrose, the NSSE 
benchmark of effective educational practices provided a point of departure for conversa¬ 
tions about learning, student engagement, and institutional improvement. As Morehouse 
(2012) aptly puts it, “Surveys provide mathematical descriptions of trends, attitudes, 
and opinions of a group of participants” (p. 78). The present study has demonstrated the 
need for an additional analysis of student engagement that would complement NSSE and 
TSSSE by providing qualitative data to supplement the quantitative data provided by the 
two survey instruments. Whereas previous studies have used qualitative data simply to 
triangulate the quantitative data to affirm or disprove a particular NSSE benchmark (Co¬ 
hen, Manion, & Morrison, 2000), the qualitative data from the focus groups in the present 
study provided—in the students’ own words—insight into the “why” of their lived educa¬ 
tional experiences.^ 


Notes 

1 The Theological School Survey of Student Engagement is an adaptation, for graduate 
students of theology, of the National Survey of Student Engagement and the Law School 
Survey of Student Engagement. For validity and reliability issues see Fuller (2009). 

2 NSSE has recently replaced the term “benchmarks” with “engagement indicators” and 
revised the categories. 
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